Method of generating three-dimensional model, training data, machine learning model, and system

ABSTRACT

A method of generating a three-dimensional model of an object, is executed by a processor. The method includes executing rendering of the three-dimensional model of the object based on an image captured by the imaging device; and modifying the three-dimensional model.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present U.S. non-provisional application is a continuationapplication of and claims the benefit of priority under 35 U.S.C. §365(c) from PCT International Application PCT/JP2019/039868 filed onOct. 9, 2019, which is designated the U.S., and is based upon and claimsthe benefit of priority of Japanese Patent Application No. 2018-191935,filed on Oct. 10, 2018, the entire contents of which are incorporatedherein by reference.

TECHNICAL FIELD

The present disclosure may relate to a three-dimensional scanningdevice, a method of generating a three-dimensional model, training data,and a machine learning model.

SUMMARY

According to one aspect of an embodiment of the present inventiveconcept, a method of generating a three-dimensional model of an object,is executed by a processor. The method may include executing renderingof the three-dimensional model of the object to be scanned based on atleast one image captured by the imaging device; and modifying thethree-dimensional model.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram illustrating a general configuration of athree-dimensional scanning device according to an embodiment;

FIG. 2 is a schematic diagram illustrating the bottom surface side of asupport table in FIG. 1 ;

FIG. 3 is a flow chart illustrating steps of three-dimensional scanningprocessing executed by a three-dimensional scanning device according toan embodiment;

FIG. 4 is a diagram illustrating an example of a three-dimensional modelgenerated in three-dimensional scanning processing; and

FIG. 5 is a schematic diagram illustrating a general configuration of athree-dimensional scanning device according to a modified example of anembodiment.

EMBODIMENTS OF THE INVENTION

According to the present disclosure, a novel method of three-dimensionalscanning can be provided.

In the following, an embodiment will be described with reference to theaccompanying drawings. To facilitate understanding of the description,the same elements have the same reference numerals assigned throughoutthe drawings as far as possible, to omit duplicated description.

In the drawings, the x-direction, the y-direction, and the z-directionare perpendicular to one another. The z-direction is a directioncorresponding to the axis of rotation of a support table 2, and istypically the vertical direction. The positive side in the z-directionis designated as the upper side, and the negative side in thez-direction is designated as the lower side. The x-direction and they-direction correspond to extending directions of a top surface 2A ofthe support table 2, and are typically horizontal directions. Note thateach of the terms “vertical direction”, “perpendicular direction”,“horizontal direction”, and the like used in the present embodiment,includes one direction associated with the term and directions within apredetermined range that includes the one direction.

With reference to FIG. 1 and FIG. 2 , a configuration of athree-dimensional scanning device 1 according to an embodiment will bedescribed. FIG. 1 is a schematic diagram illustrating a generalconfiguration of a three-dimensional scanning device 1 according to theembodiment; and FIG. 2 is a schematic diagram illustrating the bottomsurface 2B side of a support table 2 in FIG. 1 .

The three-dimensional scanning device 1 may be a device that scans anobject to be scanned 10 to capture multiple image data items of theimaged object to be scanned 10, and synthesizes the image data items, togenerate a three-dimensional model. The object to be scanned 10 includesobjects in any shape and having a size that can be placed inside ARmarkers 6 of the support table 2, for example, a container such as a cupas exemplified in FIG. 1 . Also, the scanning direction of the object tobe scanned 10 may be, for example, a circumferential direction aroundthe z direction as the central axis. Note that the object to be scanned10 is not limited to having a size that can be placed inside the ARmarkers 6 of the support table 2.

As illustrated in FIG. 1 , the three-dimensional scanning device 1 mayinclude the support table 2 (support part), upper cameras 3A and 3B(second image sensors), lower cameras 4A and 4B (first image sensors), aturning part 5, the AR markers 6 (relative position indicators), acontrol device 7, and lights 8A and 8B.

The support table 2 may be a pedestal on which the object to be scanned10 is placed, to turn around the placed object to be scanned 10. Thesupport table 2 may be, for example, a transparent plate such as anacrylic plate. The top surface 2A of this transparent plate may be aplacement surface on which the object to be scanned 10 is placed. Also,as the support table 2 is a transparent plate, the object to be scanned10 placed on the top surface 2A can be imaged (hereafter, also referredto as “visual recognized”) from a bottom surface 2B as the backside ofthe placement surface as illustrated in FIG. 2 . Note that the supporttable 2 may be formed of a material that is translucent orlight-permeable to a certain extent, as long as the object to be scanned10 can be imaged, or information on the object to be scanned 10 can beobtained from the bottom surface 2B side.

The upper cameras 3A and 3B may be installed over the top surface 2Aside of the support table 2, and can capture images of the object to bescanned 10 when being oriented toward the object to be scanned 10 inobliquely upward directions with respect to the placement surface 2A.The upper cameras 3A and 3B may be installed at different tilt angles.Note that in the present embodiment, the “obliquely upward directionwith respect to the placement surface 2A” is a direction extending abovethe placement surface 2A and tilted outward with respect to the verticalaxis at the center at which the object to be scanned 10 is placed on thesupport table 2. In other words, each of the upper cameras 3A and 3Baccording to the present embodiment obtains information in a directionother than a direction right above or right beside the object to bescanned 10.

The lower cameras 4A and 4B may be installed on the bottom surface 2Bside of the support table 2, and can captured images of the object to bescanned 10 when being oriented toward the object to be scanned 10 inobliquely downward directions with respect to the placement surface 2A.The lower cameras 4A and 4B may be installed at different tilt angles.Note that in the present embodiment, the “obliquely downward directionwith respect to the placement surface 2A” is a direction extending belowthe bottom surface 2B of the support table 2 and tilted outward withrespect to the vertical axis at the center of the support table 2. Inother words, each of the lower cameras 4A and 4B according to thepresent embodiment obtains information in a direction other than adirection right below the object to be scanned 10.

The upper cameras 3A and 3B and the lower cameras 4A and 4B are examplesof image sensors that capture images of the object to be scanned 10. Theupper cameras 3A and 3B and the lower cameras 4A and 4B may be, forexample, RGB-D cameras that are capable of capturing RGB images anddepth images of the object to be scanned 10. A depth image may includeinformation on depth (depth) to the object to be scanned 10. Note thatin the following, the upper cameras 3A and 3B may be also collectivelyreferred to as the “upper cameras 3”, and the lower cameras 4A and 4Bmay be also collectively referred to as the “lower cameras 4”.

In the present embodiment, the “image sensor” may mean an element thatcan capture RGB images and/or depth images of the object to be scanned10, and may include all camera devices such as RGB-D cameras and depthcameras, sensors such as CMOS sensors and depth sensors built intocamera devices, and sensors used as stand-alone elements. Also, the“image sensor” may be configured to obtain point cloud information by alaser scanner.

The turning part 5 according to the present embodiment may turn thesupport table 2 around approximately the center of the support table 2.An existing power transmission system can be used for the mechanism ofthe turning part 5. For example, the turning part 5 may be configured toinclude a motor and a gear mechanism so as to transfer the driving forceof the motor to the axis of rotation of the support table 2 through thegear mechanism, or may be configured to turn the support table 2 aroundby giving the driving force to the outer edge of the support table 2.

The AR markers 6 are an example of indicators that indicate the relativepositions of the object to be scanned 10 with respect to the uppercameras 3A and 3B and the lower cameras 4A and 4B. In the presentembodiment, the AR markers 6 are arranged on the top surface 2A and thebottom surface 2B of the support table 2 around the placement positionof the object to be scanned 10.

The AR markers 6 may be images of predetermined patterns serving aslabels in an image recognition-type AR (Augmented Reality) system. TheAR markers 6 may be typically rectangular, and may have two-dimensionalbit patterns in two colors of black and white, and marker IDs can berecognized by simple image recognition. In the present embodiment, theAR markers 6 having a flat surface are used, and the distance, angle,and the like with respect to a camera are calculated from the shapes,such as the degree of distortion of images when captured with thecamera; also, the AR markers 6 may be used in a way so as to display3DCG or the like at the marker positions, based on the obtainedinformation.

Also in the present embodiment, as illustrated in FIG. 1 , the ARmarkers 6 have rectangular-shaped planar bit patterns. The multiple ARmarkers 6 arranged around the object to be scanned 10 may have bitpatterns different from one another, and may record, for example,information on angles of rotation from the reference markers 6A and 6Bserving as reference positions in the turning direction. The referencemarkers 6A and 6B may be markers at the reference positions among thegroup of AR markers arranged on the top surface 2A and on the bottomsurface 2B, respectively, and may have approximately the same positionin the circumferential direction.

The control device 7 may control operations of the three-dimensionalscanning device 1. The control device 7 according to the presentembodiment may control scanning of the object to be scanned 10 executedby the upper cameras 3A and 3B and the lower cameras 4A and 4B. Also,the control device 7 may generate a three-dimensional model of theobject to be scanned 10, based on captured image data obtained by thescanning. As functional units related to these functions, the controldevice 7 may include an imaging control unit 71, a relative positioncalculation unit 72, and a model generation unit 73.

The imaging control unit 71 according to the present embodiment maycontrol operations of the turning part 5, and the upper cameras 3A and3B and the lower cameras 4A and 4B, to scan the object to be scanned 10by the upper cameras 3A and 3B and the lower cameras 4A and 4B, duringthe turning executed by the turning part 5, so as to capture multipleimages (RGB images and depth images in the present embodiment).

Based on images of the AR markers 6 in the images captured by the uppercameras 3A and 3B and the lower cameras 4A and 4B, the relative positioncalculation unit 72 according to the present embodiment may calculateinformation on relative positions of the object to be scanned 10 withrespect to the upper cameras 3A and 3B and the lower cameras 4A and 4B,corresponding to the respective items of the image information capturedby the imaging control unit 71. Here, the relative position may includenot only information on the coordinates of each camera, but alsoinformation on the attitude such as the tilt of the optical axis of thecamera.

The model generation unit 73 according to the present embodiment maygenerate a three-dimensional model of the object to be scanned 10, basedon the information on multiple images captured by the imaging controlunit 71, and the information on the relative positions calculated by therelative position calculation unit 72.

Part or the entirety of the control device 7 may be implemented by anyhardware components, software components, or a combination of these. Thecontrol device 7 may be primarily constituted with a microcomputer thatincludes, for example, one or more processors such as a CPU (CentralProcessing Unit), a RAM (Random Access Memory), a ROM (Read-OnlyMemory), an auxiliary storage device, and I/O (Input-Output) interfaces,to implement the various functions described above by executing variousprograms stored in the ROM and/or the auxiliary storage device on theprocessor.

The control device 7 may include multiple instance of the samecomponents described above. Also, the control device 7 may beimplemented, for example, on a single computer; or software may beinstalled on multiple computers, to cause the multiple computers toexecute the same or different parts of processing of the software. Inthis case, a form of distributed computing may be adopted, in which eachof the computers communicates via a network interface or the like toexecute processing. In other words, the control device 7 may beconfigured as a system that implements the functions by one or morecomputers executing instructions stored in one or more storage devices.Also, the control device 7 may also be configured to process informationtransmitted from a terminal by one or more computers on the cloud, totransmit the result of the processing to the terminal.

Alternatively, the control device 7 may be configured by informationprocessing of software (programs) executed by a CPU, GPU, or the like.In the case of being configured with information processing by software,software that implements at least some of the functions of the devicesin the embodiment may be recorded on a non-transitory recording medium(non-transitory computer-readable medium) such as a flexible disk, aCD-ROM (Compact Disc Read-Only Memory), a USB (Universal Serial Bus)memory, or the like, and loaded into a computer to execute theinformation processing by the software. Also, the software may bedownloaded via a communication network. Further, information processingmay be executed by hardware by implementing the software in a circuitsuch as an ASIC (Application Specific Integrated Circuit), an FPGA(Field Programmable Gate Array), or the like.

The type of recording medium to record the software is not limited. Therecording medium is not limited to a attachable/detachable medium suchas a magnetic disk, an optical disk, or the like, and may be a fixedrecording medium such as a hard disk, a memory, or the like. Also, therecording medium may be provided in the computer, or may be providedexternal to the computer.

The lights 8A and 8B according to the present embodiment are arranged onthe top surface 2A side and on the bottom surface 2B side of the supporttable 2, respectively, to illuminate the object to be scanned 10. Thelights 8A, 8B may be arranged in accordance with the installationpositions of the upper cameras 3A and 3B and the lower cameras 4A and4B, so as to eliminate shading on the surface of the object to bescanned 10, particularly in images captured by the upper cameras 3A and3B and the lower cameras 4A and 4B.

Next, with reference to FIG. 3 and FIG. 4 , a method ofthree-dimensional scanning according to an embodiment will be described.FIG. 3 is a flow chart illustrating steps of three-dimensional scanningprocessing executed by a three-dimensional scanning device 1 accordingto the embodiment. FIG. 4 is a diagram illustrating an example of athree-dimensional model generated in three-dimensional scanningprocessing. Processing of a flow chart in FIG. 3 is executed by thecontrol device 7.

At Step S1, at the outset, the object to be scanned 10 may be placed onthe top surface 2A of the support table 2. It is favorable that theplacement position of the object to be scanned 10 includes the pivot ofthe support table 2. In the present embodiment, as illustrated in FIG. 1, the top surface 2A may be approximately circular, and may have themultiple AR markers 6 arranged at predetermined intervals on a circlethat is approximately concentric with the outer edge of the top surface2A; therefore, it is favorable that the placement position of the objectto be scanned 10 is located at the center of the top surface 2A.

At Step S2, the imaging control unit 71 of the control device 7 maydrive the turning part 5 to start a turning operation of the supporttable 2.

At Step S3 (image capturing step), the imaging control unit 71 mayactivate the upper cameras 3A and 3B and the lower cameras 4A and 4B, tocapture images of the object to be scanned 10 both on the upper side andon the lower side, so as to obtain RGB images and depth images by theupper cameras 3A and 3B and the lower cameras 4A and 4B. The imagingcontrol unit 71 may cause the upper cameras 3A and 3B and the lowercameras 4A and 4B to capture images at the same timing while the objectto be scanned 10 is being turned around, and thereby, for example, asillustrated in the upper row in FIG. 4 , RGB images I1 to I4 of therespective cameras can be obtained simultaneously when the object to bescanned 10 is located at any position among angles of rotation. Also, atthis time, the imaging control unit 71 may also obtain depth images D1to D4 of the respective cameras. The depth images D1 to D4 may bepresented by point clouds in which pixels are colored depending on thedistance from the camera to the surface of the object to be scanned 10,for example, as illustrated in the middle row in FIG. 4 . In the examplein FIG. 4 , the depth images D1 to D4 are presented as darker as thedistance is longer, and as brighter as the distance is shorter.

At Step S3, the imaging control unit 71 may continue image capturing toobtain images while the turning part 5 turns the object to be scanned 10once. This enables the upper cameras 3A and 3B and the lower cameras 4Aand 4B to capture images of the object to be scanned 10 at multiplepositions along the turning direction, from the respective directionstilted at different angles with respect to the placement surface 2A, toobtain RGB images I1 to I4 and depth images D1 to D4 at the multiplepositions. In other words, the imaging control unit 71 can scan theobject to be scanned 10 simultaneously from the multiple directionsusing the multiple cameras. The imaging control unit 71 outputs theobtained images to the relative position calculation unit 72 and themodel generation unit 73.

At Step S4, the imaging control unit 71 may cause the turning part 5 tostop the turning operation of the support table 2.

At Step S5 (relative position calculation step), the relative positioncalculation unit 72 of the control device 7 may calculate the relativepositions of the upper cameras 3A and 3B and the lower cameras 4A and 4Bwith respect to the object to be scanned 10, based on images of the ARmarkers 6 in the RGB images I1 to I4.

The relative position calculation process at Step S5 will be described,taking the upper camera 3A as an example; the relative positioncalculation unit 72 may read, for example, information recorded on an ARmarker 6 as an indicator reflected in the RGB image I1, to detect theangle of rotation of the marker with respect to the reference marker 6A.Also, the AR marker 6 may have a rectangular bit pattern in the presentembodiment; therefore, the distance and angle of the upper camera 3Awith respect to this AR marker 6 may be calculated according to a changein shape such as the degree of distortion in the image of this AR marker6. Then, by integrating these information items, the position of theupper camera 3A with respect to the object to be scanned 10 can becalculated. As illustrated in FIG. 7 , in the case where multiple ARmarkers are captured in the RGB image I1, for example, by calculatingthe relative position for each marker 6, and taking an average,information on the relative positions of the upper camera 3A may befinally calculated, based on information on the calculated multiplerelative positions. Note that the calculation method of the relativepositions of the other upper camera 3B and the lower cameras 4A and 4Bmay be substantially the same as described above. The relative positioncalculation unit 72 may output the information on the calculatedrelative positions to the model generation unit 73.

At Step S6 (model generation step), based on the information on therelative positions of the upper cameras 3A and 3B and the lower cameras4A and 4B with respect to the object to be scanned 10 calculated at StepS5, the model generation unit 73 of the control device 7 may synthesizethe depth images D1 to D4 of the object to be scanned 10 at therespective positions in the turning direction obtained at Step S3, togenerate a three-dimensional model of the object to be scanned 10. Forexample, as illustrated in the lower row in FIG. 4 , by generating athree-dimensional mesh model based on the depth images D1 to D4, andpasting a texture generated based on the RGB images I1 to I4 on thesurface of the mesh model, a three-dimensional model with patterns andcolors can be generated. The model generation unit 73 can storeinformation on the generated three-dimensional model in the RAM and/orthe auxiliary storage device of the control device 7. Upon completion ofthe processing at Step S6, the present control flow may end.

Note that the processing at Step S5 and the processing at Step S6 may beexecuted together, by using the RGB images I1 to I4 and the depth imagesD1 to D4 of the object to be scanned 10 at the positions in the turningdirection obtained at Step S3. In other words, in the control device 7,the functions of the relative position calculation unit 72 and the modelgeneration unit 73 may be integrated.

Next, effects of the three-dimensional scanning device 1 according tothe present embodiment will be described.

For example, in the conventional three-dimensional scanner, in order togenerate a three-dimensional model of an object to be scanned, it may benecessary to repeat scanning of the object to be scanned, by replacingthe object in various orientations on the placement table. In contrast,the three-dimensional scanning device 1 according to the presentembodiment may include the upper cameras 3A and 3B that are installed onthe top surface 2A side of the support table 2 on which the object to bescanned 10 is placed, to capture images of the object to be scanned 10in oblique directions with respect to the placement surface; and thelower cameras 4A and 4B installed on the bottom surface 2B side of thesupport table 2 to capture images of the object to be scanned 10 inoblique directions with respect to the placement surface. Then, theimaging control unit 71 of the control device 7 may scan the object tobe scanned 10 in different directions by using the upper cameras 3A and3B and the lower cameras 4A and 4B to capture multiple images, while thesupport table 2 is turning around to turn the object to be scanned 10around.

With this configuration, by placing the object to be scanned 10 once onthe placement surface and turning the object to be scanned 10 once,images of the object to be scanned 10 can be collectively captured inthe upward and downward directions, especially in the oblique directionsin which a large amount of information can be obtained, and along theentire circumference; therefore, operations of replacing and scanningthe object in a variety of positions on the placement table as requiredin the conventional three-dimensional scanner become unnecessary, andthereby, the scanning time can be reduced, and a large amount ofinformation on images can be efficiently captured within a short time.Therefore, the three-dimensional scanning device 1 according to thepresent embodiment can obtain image information on the object to bescanned 10 used for model generation faster; therefore, athree-dimensional model can be generated more rapidly. Also, thescanning can be performed by simply lacing the object to be scanned 10in the vicinity of the center of the support table 2 (at least closer tothe center side with respect to the AR markers 6), and there may be noneed to precisely determine the installation position of the object tobe scanned 10, and also, there may be no need to fix the object to bescanned 10; therefore, the scanning and model generation can be executedmore easily. As a result, the three-dimensional scanning device 1according to the present embodiment can generate a three-dimensionalmodel in every direction more easily and rapidly.

Also, in the three-dimensional scanning device 1 according to thepresent embodiment, the upper cameras 3A and 3B and the lower cameras 4Aand 4B may be RGB-D cameras; namely, include depth cameras, andinformation on images captured by the upper cameras 3A and 3B and thelower cameras 4A and 4B may include information on depth (depth) to theobject to be scanned 10 obtained by the depth cameras. Accordingly, whengenerating a model from scanned images, the depth of the object to bescanned 10 can be obtained directly from the scanned images, and aprocess of calculating the depth to the surface of the object to bescanned 10 by using the scanned images may become unnecessary;therefore, the calculation cost of the model generation process can bereduced. As a result, the three-dimensional scanning device 1 accordingto the present embodiment can generate a three-dimensional model evenfaster.

Also, in the three-dimensional scanning device 1 according to thepresent embodiment, the AR markers 6 on the top surface 2A and on thebottom surface 2B of the support table 2 arranged around the placementposition of the object to be scanned 10 may be used as relative positionindicators. The relative position calculation unit 72 may calculate therelative position of each of the cameras with respect to the object tobe scanned 10, based on images of the AR markers 6 in the RGB images I1to I4 captured by the upper cameras 3A and 3B and the lower cameras 4Aand 4B.

The relative position of each of the cameras with respect to the objectto be scanned 10 may also be obtained by using a component that measuresthe angle of rotation of the object to be scanned 10, for example, byusing an encoder for measuring the angle of rotation of the supporttable 2. However, a certain sensor for angle detection such the encodermay need to be provided separately. In the present embodiment, with theconfiguration described above, the relative position of each of thecameras with respect to the object to be scanned 10 can be calculatedfrom the captured images; therefore, it may be not necessary to providea sensor for angle detection separately, and the relative position canbe obtained with a simpler configuration.

Also, in the present embodiment, the reference marker 6A as a referenceposition among the AR markers 6 on the top surface 2A, and the referencemarker 6B as a reference position among the AR markers 6 on the bottomsurface 2B may be provided, and these may be arranged at substantiallythe same position in the circumferential direction as illustrated inFIGS. 1 and 2 . This may make the reference position common toinformation on images of the upper cameras 3A and 3B on the top surface2A side, and information on images of the lower cameras 4A and 4B on thebottom surface 2B side; therefore, a relative position calculated by therelative position calculation unit 72 from images of the upper cameras3A and 3B can be made common to that calculated from images of the lowercameras 4A and 4B. This may allow the image information on the topsurface 2A side and the image information on the bottom surface 2B sideto be synthesized as they are, without aligning positions in thecircumferential direction. Accordingly, the calculation cost of themodel generation process can be reduced, and generation of athree-dimensional model can be made faster. Also, as long as one of theAR markers 6 is captured in a camera image, the relative position can becalculated without precisely determining the installation position ofthe camera; therefore, calibration of the upper cameras 3A and 3B andthe lower cameras 4A and 4B may not be required, and the device can bemoved and installed easily.

Also, in the three-dimensional scanning device 1 according to thepresent embodiment, the support table 2 may be a transparent plate suchas an acrylic plate; therefore, the object to be scanned 10 can becaptured in images by the lower cameras 4A and 4B on the bottom surface2B side as the backside.

Also, in the three-dimensional scanning device 1 according to thepresent embodiment, the upper cameras 3A and 3B and the lower cameras 4Aand 4B may include multiple (two in FIG. 1 ) cameras set to havedifferent angles in oblique directions; therefore, a greater amount ofimage information can be used for model generation, and thereby, theaccuracy of three-dimensional model generation can be further improved.Here, “different angles in oblique directions” may mean that each of theupper camera 3A and the upper camera 3B has a distinct angle that isformed by a line segment connecting the pivot of the support table 2 atthe center on the top surface 2A with the position of the camera in theobliquely upward direction, with respect to the top surface 2A. In thecase of the lower cameras 4A and 4B, “different angles in obliquedirections” may mean that each of the lower cameras 4A and 4B has adistinct angle that is formed by a line segment connecting the pivot ofthe support table 2 at the center on the bottom surface 2B with theposition of the camera in the obliquely downward direction, with respectto the bottom surface 2B.

Also, in the three-dimensional scanning device 1 according to thepresent embodiment, lights 8A and 8B may be arranged on the top surface2A and on the bottom surface 2B sides of the support table 2,respectively, to illuminate the object to be scanned 10; therefore,shading on the surface of the object to be scanned 10 can be reduced,and hues on the surface can be uniform in an image captured by each ofthe cameras. Therefore, the surface texture of a generatedthree-dimensional model can be set closer to the real object.

With reference to FIG. 5 , a modified example will be described. FIG. 5is a diagram illustrating a schematic configuration of athree-dimensional scanning device 1A according to a modified example ofthe embodiment. As illustrated in FIG. 5 , in addition to a upper camera3 and a lower camera 4, a three-dimensional scanning device may beconfigured to include a right overhead camera 13 (a third image sensor)that is installed at a position in a direction normal to the placementsurface from the placement position of the object to be scanned 10 onthe top surface 2A of the support table 2, namely, at a verticallyupward position (in the z-axis positive direction); and a rightunderneath camera 14 (a fourth image sensor) that is installed in adirection opposite to the above normal direction, extending from thebottom surface 2B of the support table 2, namely, at vertically downwardposition (in the z-axis negative direction). It is favorable that theright overhead camera 13 and the right underneath camera 14 are alsoRGB-D cameras like the upper camera 3 and the lower camera 4.

In this way, by adopting a configuration that includes the rightoverhead camera 13 and the right underneath camera 14, even in the casewhere the object to be scanned 10 has a shape with an inner bottomsurface that may be difficult to see in an image from an obliquedirection, for example, the object to be scanned 10 being an object witha deep hollow such as a cup, the object can be reliably captured inimages taken in the overhead or underneath direction. This feature canreduce blind spots on the surface of the object to be scanned 10, andthereby, the accuracy of three-dimensional model generation can beimproved.

In the embodiment described above, as an example of the indicatorsrelated to the relative positions of the object to be scanned 10 withrespect to the upper cameras 3A and 3B and the lower cameras 4A and 4B,the exemplified configuration has the multiple AR markers 6 arrangedaround the object to be scanned 10 according to the positions in thecircumferential direction; however, other patterns such as, for example,two-dimensional bar codes may be used. Also, by using a sequence ofpseudo-random numbers such as an M-sequence, annular patterns accordingto the angles of rotation may be provided.

In the embodiment described above, although the exemplifiedconfiguration has the multiple AR markers 6 arranged around the objectto be scanned 10 at approximately uniform intervals around the entirecircumference, at least one of the AR markers 6 may need to be presentin images captured by the upper cameras 3A and 3B and the lower cameras4A and 4B; therefore, in the case of having multiple AR markers 6arranged, the markers may be arranged at non-uniform intervals.

In the embodiment described above, based on images of the AR markers 6in RGB images, in the exemplified configuration, although the relativepositions of the object to be scanned 10 with respect to the uppercameras 3A and 3B and the lower cameras 4A and 4B are calculated, therelative position may be calculated, for example, by providing anencoder arranged at the axis of rotation of the turning part 5 or thesupport table 2 to measure the angle of rotation, and using the measuredangle of rotation as a relative position indicator.

In the embodiment described above, in the exemplified configuration,although the turning part 5 turns the support table 2 around, theturning part 5 simply needs to be capable of turning the support table 2around relative to the upper cameras 3A and 3B and the lower cameras 4Aand 4B; therefore, a configuration different from the embodiment may beadopted in which the support table 2 is fixed, and the upper cameras 3Aand 3B and the lower cameras 4A and 4B are turned around the supporttable 2. In this case, the pivot and the axis of rotation are defined inaccordance with the device that turns the cameras around.

In the embodiment described above, although the exemplifiedconfiguration includes two cameras of the upper cameras 3A and 3B, andtwo cameras of the lower cameras 4A and 4B, there may be, at least, oneupper camera and one lower camera. Also, there may be three or morecameras. As the number of installed cameras increases, the accuracy of athree-dimensional model becomes higher.

In the embodiment described above, in the exemplified configuration,although the upper cameras 3A and 3B and the lower cameras 4A and 4B areRGB-D cameras that can capture RGB images and depth images, aconfiguration may be adopted in which the upper cameras 3A and 3B andthe lower cameras 4A and 4B are RGB cameras to capture only RGB images.In this case, the model generation unit 73 can calculate information ondepth of an object to be scanned, for example, by using multiple RGBimages capture by the same camera at different angles of rotation.

The present embodiment may be further provided with a rendering unitthat includes a renderer to assist generation of a three-dimensionalmodel. By providing such a rendering unit, a more accuratethree-dimensional model can be generated. Also, by using the renderer,in some cases, part of information used when constructing athree-dimensional model (e.g., information on depth) may be substituted.Also, by configuring this renderer by using a trained machine learningmodel, the accuracy of a three-dimensional model can be improved. Therendering unit may modify the three-dimensional model after the modelgeneration unit 73 has generated a three-dimensional model, or mayexecute rendering before the model generation unit 73 completes thegeneration of the three-dimensional model, namely, during a course ofgeneration.

In the embodiment described above, in the exemplified configuration, thetop surface 2A of the support table 2 constituted with a transparentplate such as an acrylic plate is used as the placement surface, namely,the rigidity of the placement surface is higher than that of the objectto be scanned 10, and the placement surface is substantially flat;however, the placement surface may need at least to allow the object tobe scanned 10 to be captured (visible) also from the backside of theplacement surface, and to be hardly visible in an image captured by acamera. In other words, the transparency of the member may simply needto reach a certain level at which information on the object to bescanned 10 can be obtained appropriately. The plate may be formed of amaterial through which an image of the object to be scanned 10 cannot becaptured entirely, but has openings in the material through which animage of the object to be scanned 10 can be captured, such as a platehaving holes pierced, a net, or the like. Also, for example, theplacement surface may be formed of a material with a rigidity lower thanthe object to be scanned 10, such as a bag-like member including amaterial having a flexible shape, such as film, vinyl, net, liquid, orthe like. In other words, the placement surface may be configured to bedeformable in accordance with the shape of a placed object to be scanned10. With this configuration, even in the case where the object to bescanned 10 is an object that deforms when placed on a soft acrylic plateat the contacting portion, for example, a stuffed toy, deformation ofthe object to be scanned 10 can be reduced when placed on the placementsurface; therefore, a three-dimensional model in a state of reduceddeformation due to the external force can be generated. If such athree-dimensional model is generated, for example, the three-dimensionalmodel of the object to be scanned 10 can be used without noticeabledifferences even in a position of the three-dimensional model that isdifferent from that taken while scanning, and thereby, the versatilityof the three-dimensional model can be improved. Also, in the case wherethe support table 2 is not a transparent plate, the support table 2 mayhave another configuration other than the pedestal (e.g., aconfiguration in which a film having a placement surface is suspendedfrom above).

As above, the present embodiment has been described with reference tospecific examples. However, the present disclosure is not limited tothese specific examples. Modifications of these specific examplesdesigned by those skilled in the art appropriately are also covered inthe scope of the present disclosure as long as the features in thepresent disclosure are provided. The elements and the arrangement,conditions, shapes, and the like of these provided in each of thespecific examples described above, are not limited to those exemplified,and can be changed appropriately. Combinations of the elements providedin the specific examples described above can be changed appropriatelyunless technical inconsistency is introduced.

The upper cameras 3, the lower cameras 4, the right overhead camera 13,and the right underneath camera 14 may be elements other than cameradevices, and may be, for example, smartphones.

In the embodiment described above, in the exemplified configuration,although the object to be scanned 10 placed on the top surface 2A of thesupport table 2 is simultaneously captured in images by using the uppercameras 3A and 3B and the lower cameras 4A and 4B, a configuration maybe adopted in which only the lower cameras 4A and 4B are used. Also, theupper cameras 3A and 3B and the lower cameras 4A and 4B may captureimages at different timings. In other words, the object to be scanned 10may be captured in images by the lower cameras 4A and 4B after theobject to be scanned 10 has been captured in images by the upper cameras3A and 3B, or may be captured in the reversed order. By provingindicators as in the present embodiment, generation of athree-dimensional model can be facilitated based on images captured atsuch different timings. Also, the indicators allows image capturing tobe executed with a single camera without preparing multiple cameras.

INDUSTRIAL APPLICABILITY

By using the three-dimensional scanning devices 1 and 1A according tothe embodiment described above, a virtual space can be generated byscanning nearby objects, to use the virtual space as a simulator formachine learning such as task learning of robots. In other words, thethree-dimensional scanning devices 1 and 1A according to the embodimentdescribed above can be applied to a scanner device to generate athree-dimensional model for such a simulator. In other words, by usingdata obtained by the method of three-dimensional scanning according tothe embodiment described above, data for machine learning or trainingdata for machine learning can be generated. This training data can alsobe used for generate a machine learning model.

In order to generate such a virtual space for a simulator, it isnecessary to scan a large number of objects to generatethree-dimensional models; therefore, one may conjecture that an enormousscanning time would be required. However, the three-dimensional scanningdevices 1 and 1A according to the embodiment described above can shortenthe scanning time as described above, and thereby, can expeditegeneration of three-dimensional models; therefore, it can be consideredadvantageous to generate a virtual space for a simulator.

The invention claimed is:
 1. A method of generating a three-dimensionalmodel of a real object, the method comprising: imaging the real objectplaced on a transparent plate for which the three-dimensional model isto be generated and a first marker by at least a lower image sensor thatis installed below the transparent plate from below the real object tocapture an image including the real object and the first marker, saidtransparent plate being rotated together with the real object and thefirst marker relative to the at least the lower image sensor, or saidlower image sensor being rotated relative to the transparent plate, thereal object and the first marker; calculating position information ofthe lower image sensor based on at least a distortion of a shape of thefirst marker in the captured image including the real object and thefirst marker; and generating the three-dimensional model based on atleast both (i) a first image, that is a part of the captured imageincluding the real object and the first marker, of the real object and(ii) the calculated position information of the lower image sensor. 2.The method as claimed in claim 1, wherein the image is RGB data.
 3. Themethod as claimed in claim 1, wherein the generating generates thethree-dimensional model, by utilizing a machine learning model havingbeen trained in such a manner that a generation accuracy of thethree-dimensional model is improved.
 4. The method as claimed in claim1, wherein the three-dimensional model includes information on a surfaceof the real object obtained from the image.
 5. The method as claimed inclaim 4, wherein the information is at least one of surface texture,pattern, or color.
 6. The method as claimed in claim 3, wherein thegenerating generates at least depths of the three-dimensional modelwithout information on depths captured from the real object, by usingthe trained machine learning model.
 7. The method as claimed in claim 3,wherein the machine learning model has been trained to generate at leastinformation on depths of the three-dimensional model without informationon depths of the real object.
 8. A method of generating training data,the method comprising: generating training data for machine learning,based on the three-dimensional model of the real object generated by themethod as claimed in claim
 1. 9. A method of generating a first machinelearning model, the method comprising: generating the first machinelearning model by using the training data for machine learning generatedby the method as claimed in claim
 8. 10. The method as claimed in claim1, wherein the imaging of the real object is performed utilizing athree-dimensional scanning device, the three-dimensional scanning deviceincluding: a transparent plate on which the real object to be imaged isplaced and the first marker and a second marker are arranged; and atleast the lower image sensor and an upper image sensor installed belowand above the transparent plate respectively, the lower image sensorimaging an underside of the real object placed on the transparent platethrough the transparent plate and the upper image sensor imaging anupper side of the real object placed on the transparent plate and thesecond marker to capture an image including the real object and thesecond marker, wherein the first image and a second image of the realobject in the image including the real object and the second marker areused to generate the three-dimensional model.
 11. The method as claimedin claim 10, the method further comprising: calculating information on arelative position of the real object, based on a plurality of imagescaptured by the lower image sensor or the upper image sensor.
 12. Themethod as claimed in claim 11, the method further comprising: generatingthe three-dimensional model of the real object, based on the pluralityof images and the information on the relative position calculated by thecalculating.
 13. The method as claimed in claim 10, wherein the lowerimage sensor installed below the transparent plate is configured tocapture color information and depth information of the underside of thereal object.
 14. The three-dimensional model generated by the method ofgenerating the three-dimensional model as claimed in claim
 1. 15.Training data for machine learning generated by using thethree-dimensional model generated by the method of generating thethree-dimensional model as claimed in claim
 1. 16. A machine learningmodel trained with training data for machine learning as claimed inclaim
 1. 17. The method as claimed in claim 10, further comprisingrotating the real object by rotating the transparent plate on which thereal object is being placed, wherein the imaging images the undersideand the upper side of the real object a plurality of times during therotating in such a manner that the underside of the rotating real objectis imaged from below by the lower image sensor from different angles andthe upper side of the rotating real object is imaged from above by theupper image sensor from different angles.
 18. The method as claimed inclaim 3, wherein the generating of the three-dimensional model includes:generating a first three-dimensional model; and generating thethree-dimensional model that has an accuracy better than the firstthree-dimensional model, based on a modifying process utilizing thegenerated first three-dimensional model and the machine learning model.19. The method as claimed in claim 1, further comprising: imaging thereal object and a second marker by at least an upper image sensor fromabove to capture an image including the real object and the secondmarker; and calculating position information of the upper image sensorbased on at least a distortion of a shape of the second marker in thecaptured image including the real object and the second marker, whereinthe generating generates the three-dimensional model based on at least(i) the first image of the real object, (ii) the calculated positioninformation of the lower image sensor, (iii) a second image, that is apart of the captured image including the real object and the secondmarker, of the real object, and (iv) the calculated position informationof the upper image sensor.
 20. The method as claimed in claim 1, whereinthe position information of the lower image sensor is information on arelative position of the lower image sensor relative to the firstmarker.
 21. The method as claimed in claim 1, wherein the first markeris arranged on the transparent plate.
 22. A system used for generating athree-dimensional model of a real object comprising: a transparent plateon which the real object is placed; a first marker and a second markerarranged on the transparent plate so that an image of the first markerand an image of the second marker are captured from below and from abovethe transparent plate by a lower image sensor and an upper image sensorrespectively, the first and second markers being configured to be usedfor calculating position information of the lower and the upper imagesensors respectively; and the lower image sensor and the upper imagesensor installed below and above the transparent plate respectively, thelower image sensor being configured to capture a first image of anunderside of the real object being placed on the transparent platethrough the transparent plate with the image of the first marker and theupper image sensor being configured to capture a second image of anupper side of the real object being placed on the transparent plate withthe image of the second marker, wherein the position information of thelower and upper image sensors and the first and second images capturedrespectively by the lower and upper image sensors are used forgenerating the three-dimensional model of the real object.
 23. Thesystem as claimed in claim 22, wherein the at least one processorconfigured to generate at least depths of the three-dimensional modelfrom the image without information on depths captured from the realobject, by utilizing a machine learning model.
 24. The system as claimedin claim 22, further comprising the at least one processor configured togenerate the three-dimensional model.
 25. The system as claimed in claim22, further comprising a controller configured to control thetransparent plate to rotate and the at least lower image sensor and theupper image sensor to capture images.
 26. The system as claimed in claim22, further comprising at least one processor configured to generate thethree-dimensional mesh model of the real object based on the positioninformation of the lower and upper image sensors and the first andsecond images captured respectively by the lower and upper imagesensors.
 27. An apparatus used for generating a three-dimensional modelof a real object, comprising: at least one memory configured to store aprogram; and at least one processor configured to execute the programto: obtain at least a first captured image including an underside of thereal object placed on a transparent plate and a first marker from belowthe real object by a lower image sensor installed below the transparentplate and a second captured image including an upper side of the realobject placed on the transparent plate and a second marker from abovethe real object by an upper image sensor installed above the transparentplate, said transparent plate being rotated together with the realobject, the first marker and the second marker relative to the lowerimage sensor and the upper image sensor, or said lower image sensor andthe upper image sensor being rotated relative to the transparent plate,the real object, the first marker and the second marker; calculate firstposition information indicating where the first captured image has beencaptured from with respect to the first marker, based on at least adistortion of a shape of the first marker in the first captured imageincluding the underside of the real object and the first marker;calculate second position information indicating where the secondcaptured image has been captured from with respect to the second marker,based on at least a distortion of a shape of the second marker in thesecond captured image including the upper side of the real object andthe second marker; and generate the three-dimensional model of the realobject, based on at least (i) an image of the underside of the realobject that is a part of the first captured image including theunderside of the real object and the first marker, (ii) the firstposition information, (iii) an image of the upper side of the realobject that is a part of the second captured image including the upperside of the real object and the second marker, and (iv) the secondposition information.
 28. The apparatus as claimed in claim 27, whereinthe first marker is arranged on the transparent plate.